Despite recent advances, goal-directed generation of structured discrete data remains challenging. For problems such as program synthesis (generating source code) and materials design (generating molecules), finding examples which satisfy desired constraints or exhibit desired properties is difficult. In practice, expensive heuristic search or reinforcement learning algorithms are often employed. In this paper, we investigate the use of conditional generative models which directly attack this inverse problem, by modeling the distribution of discrete structures given properties of interest. Unfortunately, the maximum likelihood training of such models often fails with the samples from the generative model inadequately respecting the input properties. To address this, we introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We avoid high-variance score-function estimators that would otherwise be required by sampling from an approximation to the normalized rewards, allowing simple Monte Carlo estimation of model gradients. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value. In both cases, we find improvements over maximum likelihood estimation and other baselines.

Weaknesses: The improvements over Maximum Likelihood are very moderate and no comparisons are made with more computationally expensive RL approaches (at least on the small QM9 dataset it would be interesting). It would be very interesting to see the performance tradeoff between the proposed approach and the Monte Carlo estimation of the expectation term in Eq 11. One of the promising features of generative algorithms for molecules is their supposed ability to capture a complex statistical distribution of plausible molecules that can be made, paid for, stored in a vial, etc. The approximately 100 million molecules that have been made and the couple of billions that can be confidently said to be makeable are samples from that distribution. It is not clear how much of chemical space is in that manifold.

The authors propose an RL-inspired way of fitting a conditional generative model to the training data with the aim of generating discrete structures, such as molecules, satisfying some desired properties. Unlike policy gradients in RL, the proposed algorithm does not require sampling from the model/policy, instead approximating the expectation of interest using the training data reweighted with the normalized rewards. This is done to avoid high gradient variance of policy gradient algorithms. The reviewers liked the novelty of the approach to this important problem. While the experimental results are not spectacular and there were some concerns about missing RL baselines and connections to reward-augmented ML, the author response addressed them in large part.

Despite recent advances, goal-directed generation of structured discrete data remains challenging. For problems such as program synthesis (generating source code) and materials design (generating molecules), finding examples which satisfy desired constraints or exhibit desired properties is difficult. In practice, expensive heuristic search or reinforcement learning algorithms are often employed. In this paper, we investigate the use of conditional generative models which directly attack this inverse problem, by modeling the distribution of discrete structures given properties of interest. Unfortunately, the maximum likelihood training of such models often fails with the samples from the generative model inadequately respecting the input properties.